Replication

Where are we?

Tuomas Eerola

7/2/23

1 – Credibility Crisis

  • “Why most published research findings are false” (Ioannidis, 2005)
  • Psychology has lion share of bias/misleading results (Simmons, Nelson, & Simonsohn, 2011)
  • Science: Serious concern about robustness of results
    • 100 studies replicated, about 36% produced statistically significant results (OSC, 2015)

1 – Solutions – Currently

  • Proposed solutions:
    • Go beyond null hypothesis testing, switch to CIs ✔︎
    • Open Data ✔︎, Shared code ✗
    • Open Access publishing ✔︎
    • Engage in study replication
  • Meta-research (Hardwicke, Serghiou, et al., 2019) concludes

Current State in Psychology (1)

Hardwicke et al. (2022) sampled 250 psychology articles

Current State in Psychology (2)

2 – Current State in our Field

Keyword search of of titles and abstracts + manual verification

JOURNAL N Replication
Journal of New Music Research 570 1
Music and Science 150 0
Music Perception 957 5
Musicae Scientiae 617 8
Psychology of Music 1380 2
Total 3674 16

2 –Closer look at the Specialist Journals

Sample (15%) of articles between 2017-2020 (100 articles)

Availability of N %
Materials 5 5
Protocol 3 3
Data 7 7
Analysis script 0 0
Preregistration 0 0

3 – Solutions?

References

Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S., & Ioannidis, J. P. A. (2019). Calibrating the scientific ecosystem through meta-research. Annual Review of Statistics and Its Application. https://doi.org/10.31222/osf.io/krb58
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022). Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). Perspectives on Psychological Science, 17(1), 239–251. https://doi.org/10.1177/1745691620979806
Hardwicke, T. E., Wallach, J. D., Kidwell, M., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2019). An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017). https://doi.org/10.31222/osf.io/6uhg5
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
OSC. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359 1366. https://doi.org/10.1177/0956797611417632

Workshop